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Impact Evaluation of Mathematics i-Ready Instruction for Elementary 
Grades using 2018-19 Data 


Abstract 


Curriculum Associates’ i-Ready® Instruction is a supplemental, online personalized instruction 
program available for reading and mathematics’. The Human Resources Research 
Organization (HumMRRO), in collaboration with Century Analytics, implemented a quasi- 
experimental design (QED) using 2018-19 i-Ready Diagnostic and Instruction data to evaluate 
the impact of Curriculum Associates’ mathematics /-Ready Instruction on student mathematics 
achievement at grades K—5. We hypothesized student achievement, as measured by the /- 
Ready® Diagnostic, would be higher for students using i-Ready Instruction for mathematics over 
a comparison group of students who did not use this instruction. We conducted matching to 
identify a set of comparison students demographically similar to our ij-Ready Instruction 
treatment students for each grade level. First, we stratified our sample by gender, English 
learner status, disability status, and economic disadvantage status. Next, we used propensity 
score matching to identify analytic samples of i-Ready Instruction and comparison students 
matched on baseline mathematics student achievement. Students who received the /-Ready 
Instruction and students in the comparison group were administered the mathematics i-Ready 
Diagnostic assessments. To evaluate impact, hierarchical-linear modeling (HLM) was 
conducted separately for each analytic sample with students at level 1 and school at level 2. 
Results suggest students using /-Ready Instruction with fidelity performed statistically 
significantly better on mathematics performance than students in grades K—-5 who did not use 
this instruction. The effect sizes fall at the upper end or exceed (in the case of kindergarten) the 
range for which recent research by Kraft (2019) has found is typical of education interventions. 
These findings provide support that, when used with fidelity, student use of i-Ready Instruction 
for mathematics is tied to higher student mathematics achievement. 


Introduction 


Founded in 1969, Curriculum Associates provides a variety of educational products and 
services with the goal of improving education for students and teachers. Two Curriculum 
Associates products include i-Ready® Diagnostic (available for K-12) and i-Ready® Instruction 
(available for K-8). The /-Ready Diagnostic assessments (a) are online, computer-adaptive 
assessments that pinpoint student needs at the sub-skill level and (b) help monitor the extent to 
which students are on track to achieve end-of-year targets. The i-Ready Diagnostic 
assessments are independent measures often used by educators as classroom benchmark 
assessments. They can be used with or without /-Ready Instruction. We provide additional 
information on the validity and reliability of the i-Ready Diagnostic as a measure of student 
achievement in our methodology discussion below. i-Ready Instruction is a supplemental 
program that provides online, individualized instruction adjusted to student needs. 


The Human Resources Research Organization (HumMRRO) is an independent research 
organization that specializes in program evaluation and quantitative methodology. Century 
Analytics is a small business with various education research expertise including quasi- 
experimental design and What Works Clearinghouse (WWC) standards. 


1 https:/Awww.curriculumassociates.com/products/i-ready 


Impact Evaluation of Mathematics i-Ready Instruction for Elementary Grades using 2018 — 19 Data 1 


PS HuMRRO 


HumRRO and Century Analytics conducted an evaluation to examine the impact of i-Ready 
Instruction on mathematics achievement for students in elementary grades K—5 using 2018-19 
data. This was one in a series of evaluations examining the impact of Curriculum Associates’ 
interventions on student achievement. This study was designed to meet the required rigor of the 
WWC 4.0 standards to achieve a rating of Meets WWC Group Design Standards with 
Reservations (WWC, 2017a), and to meet guidelines for a Level 2 (or Moderate) rating for the 
Every Student Succeeds Act (ESSA) guidance for evidence-based research (U.S. Department 
of Education, 2016). To accomplish this, we used a quasi-experimental design (QED), 
established baseline equivalence between the treatment and comparison groups, included 
baseline achievement as a covariate, and used a sampling design that mitigates the effects of 
any confounding factors. 


There were key differences between this study and past studies. Specifically, previous studies 
considered school as the unit of /-Ready Instruction assignment, whereas this study considered 
student as the unit of assignment. This change in unit of assignment acknowledges the inherent 
flexibility of i-Ready Instruction implementation. For example, some schools may implement at 
the school-level, the grade-level, or the classroom-level, while other schools may implement /- 
Ready Instruction at the individual student-level so they can target specific groups of students. 
In addition, our past studies included only schools using /-Ready Diagnostic and Instruction, or i- 
Ready Diagnostic only for the comparison group, with general education students. Thus, those 
schools using /-Ready Diagnostic (with or without /nstruction) with select subsets of students 
were removed from our sample. Because our data support various types of implementation 
occurring across schools, and we understand it is Curriculum Associates intent that these 
different implementations are valid uses, this study includes students from schools that are 
implementing i-Ready Diagnostic with or without /nstruction in a variety of ways. 


Defining i-Ready Instruction 


The impact of /-Ready Instruction on student achievement was the focus of this evaluation. /- 
Ready Instruction is an online personalized instruction program aligned to college- and career- 
ready standards that includes engaging multimedia instruction and progress monitoring into 
online lessons. Lessons are intended to provide a consistent best-practice lesson structure and 
build students’ conceptual understanding. /-Ready Instruction is intended to be used in 
conjunction with i-Ready Diagnostic which monitors student progress and identifies student 
performance in reading and mathematics. This diagnostic information helps target student- 
specific intervention, which can be provided through /-Ready Instruction. 


Curriculum Associates developed a Theory of Action (TOA) that features the key 
implementation components of i-Ready Instruction, the intended intermediate outcomes, and 
the intended long-term outcomes. The key implementation components highlight actions 
recommended by students, teachers, and leaders to obtain the long-term outcome of improved 
student learning in reading and mathematics. Among others, the key components include 
support at the school and district leadership levels, monitoring of student progress by teachers, 
and student use of i-Ready Instruction to work through a personalized, scaffolded instruction 
path. The /-Ready Instruction TOA is provided in Appendix A. 


Curriculum Associates provides guidance to districts and schools on how to implement /-Ready 
Instruction to best benefit student learning (Curriculum Associates, 2019). Guidance indicates 
students achieve greater gains when using /-Ready Instruction for an average of at least 30 
minutes per week, per subject area. In addition, Curriculum Associates recommends use for 12 
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to 18 calendar weeks between two administrations of the i-Ready Diagnostic (Curriculum 
Associates, 2018). 


Research Questions 


The purpose of this study was to determine the impact of i-Ready Instruction on student 
achievement in mathematics. We examined the following key research question separately for 
each grade K—5 of our study: 


Do students who use /-Ready Instruction for mathematics have higher mathematics 
achievement as measured by the i-Ready Diagnostic than students who use /-Ready 
Diagnostic only? 


We hypothesized that student achievement for mathematics would be higher for students who 
used /-Ready Instruction with fidelity, based on the criteria described in the TOA and user 
guidance (Curriculum Associates, 2019). Our hypothesis was based on the belief that students 
benefit from the /-Ready Instruction targeted to their specific needs in mathematics. 


Methodology 


In this section, we describe the methodology for conducting our impact analysis. We begin with 
initial design decisions. We then discuss the student selection and matching process as well as 
our analytic model and examination of baseline equivalence. Finally, we discuss our impact 
analysis results. 


Initial Design Decisions 
Cluster-Level Design 


We used the student as the unit of assignment for this study to acknowledge the flexibility 
intended by /-Ready Instruction and to include students from schools with various 
implementation types. Matching was conducted at the student-level and, thus, the analytic 
model examined the outcome at the student-level. However, we also considered potential 
influence of school-level factors and thus decided to include a two-level analytic model with 
school characteristics at level 2 and students at level 1. 


Baseline and Outcome Measure 


We selected the i-Ready Diagnostic as both the baseline and outcome measure for all students 
participating in this study (i.e., -Ready Instruction students and comparison group students). /- 
Ready Diagnostic for mathematics measures achievement aligned to common mathematics 
content and skills with demonstrated test score reliability. Marginal reliabilities range from 0.92 
to 0.96 and test-retest reliabilities range from 0.71 to 0.86 for mathematics through grade 5. 
Therefore, this assessment meets the WWC 4.0 standards for an acceptable baseline and 
outcome measure (WWC, 2017a). 


The i-Ready Diagnostic assessments align to college- and career-ready standards so that 
results can inform student placement decisions, offer explicit instructional advice, and prescribe 
resources for targeted instruction and intervention. The assessments are used by some schools 
and districts in conjunction with i-Ready Instruction and by others as a stand-alone diagnostic 
assessment without the use of /-Ready Instruction. The i-Ready Diagnostic assessments for 
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mathematics and reading are currently used by more than 6.5 million students across the United 
States. Thus, the use of i-Ready Diagnostic as the outcome measure allowed us to include a 
large sample of students from across the United States. The /-Ready Diagnostic is intended to 
be administered in a standardized manner across schools (Curriculum Associates, 2019b). 
Specifically, teachers are to schedule the first (fall) Diagnostic 2-3 weeks into the school year in 
two 45—50-minute sessions. Teachers also are encouraged to test technology to ensure proper 
function and have pencils and paper available as scratch paper. Test administrators provide 
instructions to their students and motivate them to do their best. Teachers monitor students as 
they complete the assessments. 


Multiple studies have been conducted to support the reliability and validity of the mathematics /- 
Ready Diagnostic as well as its consistency with education standards used across the United 
States. Since being released in summer 2011, /-Ready Diagnostic has been reviewed and 
approved at the state level as an assessment, instructional resource, or intervention in Arizona, 
California, Colorado, Connecticut, Delaware, Florida, Georgia, Idaho, Indiana, Massachusetts, 
Mississippi, Nevada, New Mexico, New York, North Carolina, Ohio, Oklahoma, Oregon, 
Tennessee, Utah, and Virginia. 


Curriculum Associates has conducted multiple linking studies examining /-Ready Diagnostic 
scores for mathematics at grades 3-8 that provide evidence the /-Ready Diagnostic measures 
skills consistent with student expectations and can be used as a student mathematics 
achievement measure. For example, a study using 2016 data examined the correlation between 
i-Ready Diagnostic and the Smarter Balanced summative assessments, the Partnership for 
Assessment of Readiness for College and Careers (PARCC), and state testing programs in 
Florida, Georgia, Indiana, Michigan, Mississippi, New York, North Carolina, Ohio, and 
Tennessee. These studies show strong correlations between /-Ready Diagnostic scores and 
scores on these national and state tests. The average correlations across grades between the /- 
Ready Diagnostic for mathematics ranged from 0.82 (North Carolina End-of-Grade 
assessments) and 0.88 (Smarter Balanced and Michigan M-STEP). These studies also provide 
evidence that the i-Ready Diagnostic content is highly consistent with what students across the 
United States are expected to learn (Curriculum Associates, 2019). Curriculum Associates 
recently completed linking studies for Colorado, Kentucky, and Missouri. In addition, Curriculum 
Associates has commissioned Odell Education and others to complete alignment studies to 
demonstrate the degree of alignment between the content on i-Ready Diagnostic and current 
sets of state standards. Specifically, they have conducted alignment studies for the Common 
Core State Standards (CCSS), and for the Florida, Indiana, Louisiana, Michigan, Ohio, and 
South Carolina state standards. 


Required Number of Students 


We conducted power analyses using Optimal Design software (Spybrook et al., 2011) to identify 
the total number of students required at each grade level to reject the null hypothesis that there 
is no difference in student mathematics achievement between the treatment and comparison 
group. Statistical power is influenced by various factors. We used data from previous studies 
HumRRO conducted using /-Ready Diagnostic as an outcome to estimate conservative and 
optimistic parameters for use in the power analysis. These parameters were: (a) 0.90 for the 
relationship between the baseline and outcome variable, and (b) 0.10 and 0.30 for the intraclass 
correlation coefficient (ICC). Results of the power analyses indicated sample sizes of a 
minimum of 400 students would be sufficient to reach our desired statistical power of 0.80. This 
level of statistical power provides an 80% chance of detecting a statistically significant 
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difference with 95% confidence, if one exists. Our student samples across all grades far 
exceeded the minimum. 


Analytic Model 


Our model for the impact analyses incorporated student- and school-variables. The baseline 
difference model used to estimate baseline equivalence for our matched sample was based on 
the impact model. As previously discussed, we chose a two-level model with level 1 as the 
student and level 2 as school. 


Impact Model 


We used HLM to estimate the impact of i-Ready Instruction on student achievement. We 
included the following student-level covariates in each analysis: 


e Group membership (0 = comparison; 1 = treatment) 
e ji-Ready Diagnostic mathematics baseline performance (grand mean centered) 


e Blocking variables (i.e., dummy codes) to account for strata used in matching (described 
in the matching section of this report) 


Although we considered the student to be our unit of assignment, with the understanding that 
many schools intentionally do not use /-Ready Instruction with all students, we also wanted to 
capture and control for potential school-level factors. We were especially interested in 
identifying variables that would provide unique information from the student-level variables. We 
used the following school-level covariates in each analysis: 


e Traditional school indicator (0 = K—5 structure; 1 = other) 

e Location (town, suburban, rural, city) 

e Charter/magnet school indicator (0 = not charter or magnet; 1 = charter or magnet) 
e Percent white students 


e Percent of students eligible for free and reduced price lunch (FRL) 


Our Level 1 model described the relationship between student outcomes, student-level 
characteristics, the baseline covariate, and the strata used for matching. This model level also 
included the treatment indicator. We specified level 1 of the model as follows: 


Yij = BOj + B1j(GROUPij) + B2j(PREij — PRE..) + ZBq(STRATA\) + eij 


Where Yij is the outcome for student / in school j. BO/ is the adjusted mean outcome for 
comparison students in school j. B1/ is the adjusted mean difference in outcome due to the 
student’s group membership (i.e., the treatment effect), and GROUP is an indicator variable 
coded 1 for students in the /-Ready Instruction group and 0 for students in the comparison 
group. B2j is the adjusted difference in outcome due to the student’s baseline achievement 
score (grand mean centered). Bq is a vector of blocking variables to account for the strata used 
in matching. ei is the random error in the achievement outcome associated with student / in 
school j not accounted for in the model. 


We specified level 2 of the model as follows: 
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BOj = yOO + yO1(STRUCTURE)) + y02(CHARTER)) + y03(PERWHITE;) + y04(PERFRL,) 
+ Zyk(LOCATION)) + u0j 


B1j = y10 
B2j = y20 
ZBp = ypO 
ZBq = yqO 


Where y00 is the grand mean. y01 is added to control for school grade-level structure where 
STRUCTURE is coded as 0 for schools with a typical grade level structure (K—5 for elementary 
school) and 1 for schools with an atypical grade structure. y02 is the additive effect for charter or 
magnet schools. yO3 and y04 are added to control for school characteristics of percent white 
and percent FRL, respectively. Zyk is a vector of three dummy variables to control for school 
location. uOj is the random error in the achievement outcome associated with school j. The 
regression slopes for the treatment, student baseline achievement, student demographics and 
strata are fixed across schools. 


Baseline Difference Model 


We used the model below to estimate the baseline difference between students in the treatment 
group and the comparison group. This model follows the same structure as the impact analysis 
model but excludes covariates. 


We specified level 1 of the model as: 
Yij = BOj + B1s(GROUPY) + ZBq(STRATAY) + ej 


Where Yij is the baseline for student / in school j. BO/ is the adjusted mean outcome for 
comparison students in school j. B1/ is the adjusted difference in outcome due to the student’s 
study group membership (i.e., the baseline difference), and GROUP is an indicator variable 
coded 1 for students in the i-Ready Instruction group and 0 for students in the comparison 
group. Bq is a vector of blocking variables to account for the strata used in matching. ejj is the 
random error in the achievement outcome associated with student / in school / not accounted for 
in the model. 


We specified level 2 of the model as: 
BOj = yOO + u0/ 
B1j = y10 
ZBq = yqO 


Identifying a Student Sample 
Defining Eligibility 


For each grade, we started with a student-level /-Ready usage file of mathematics i-Ready 
Diagnostic and i-Ready Instruction use in 2018-19 for students who had at a minimum fall and 
spring i-Ready Diagnostic scores. We next filtered to include only public school students, which 
included traditional public schools and public charter and magnet schools. This ensured we 
were including only students in a relatively traditional school environment with expectations to 
follow state adopted college and career ready standards. 
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We also filtered our sample based on availability of student-level demographic variables that 
were identified for inclusion in matching and the impact analysis model. Only students with 
available demographic data for (a) gender, (b) English learner (EL) status, (c) special education 
status, and (d) economic disadvantage status were included. We conducted data checks prior to 
removing schools that indicated students with available demographic data were not different on 
academic achievement, as measured by the /-Ready Diagnostic, than those who did not have 
demographic data. These checks provided assurance that data were missing at random. 
However, we also note that users of the /-Ready products tend to be of higher percentage 
minority and low income schools compared to all United States schools; thus, though we were 
confident our student sample used for matching was academically representative of the public 
school students using i-Ready Diagnostic or i-Ready Diagnostic and Instruction, we do not 
expect they are representative of all students in the United States. 


In addition, for a student to be eligible for the treatment group, they must have used /-Ready 
Instruction for mathematics a minimum of 18 distinct weeks for an average of at least 30 
minutes per week (Curriculum Associates, 2018). This was consistent with guidance on the 
minimum /-Ready Instruction usage at the student-level for attaining intended goals of improved 
student mathematics achievement. These students also needed to have attended a school that 
began using /-Ready Instruction to some extent prior to the 2018-19 school year. This 
requirement is based on the understanding that /-Ready Instruction implementation requires a 
start-up time to learn the technology and adjustments to scheduling before i-Ready Instruction is 
fully up and running. To be eligible for the comparison group, students must not have used any 
i-Ready Instruction for mathematics in 2018-19. We removed students not meeting the 
treatment or comparison eligibility requirements from the datafile used in matching. 


Matching 


We conducted matching at the student-level using a multi-step process. Matching was 
conducted separately by grade (K—5). Thus, we conducted each matching step six separate 
times to identify six analytic samples (i.e., six grades). 


First, we stratified our sample by gender, EL status, special education status, and economic 
disadvantage status. This assured that students were only matched to students with identical 
demographic characteristics on these four variables. The variables were selected because they 
are known to be related to student achievement (Hanover Research, 2014; van Langen, Bosker, 
& Dekkers, 2006) and were available through the /-Ready usage datafiles. This stratification 
resulted in 16 strata at each grade. Each stratum contained treatment and comparison students. 
In some strata, the treatment group was larger than the comparison group, or vice versa. Within 
each stratum, we used logistic regression to compute a propensity score for each student (Guo 
& Fraser, 2010). The propensity scores predicted the chance a student belonged to the group 
(treatment or comparison) with the smallest number of students, indicated by a value ranging 
between 0 and 1, based on the fall -Ready Diagnostic scores. We used the propensity scores 
to match each student from the smallest group (treatment or comparison) to a student from the 
largest group. We matched using the nearest neighbor method without replacement (Stuart, 
2010). Once matching was conducted for all strata within a grade, we combined the data from 
all strata into one analytic sample. 


Following specification of our analytic and baseline difference models, we removed an average 
of 7.6% of students across the six analytic samples who had incomplete data on the school- 
level variables included in the impact model. This resulted in unequal numbers of students in 
comparison and treatment groups. Figure 1 summarizes the demographic makeup of the final 
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set of students in each analytic sample. The counts of students included in each group can be 
found in Table 1 on page 11. As shown, the stratification process used in matching ensured the 
i-Ready Instruction and comparison groups were highly similar on the key demographic 
variables, despite the need to remove a small percentage of the sample to account for missing 


school-level variables. 
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Although our sampling focused on the student-level, to gain additional understanding of where 
our student-sample was from, we examined the distribution of students across urbanicity 
categories, as defined through school-level variables of the National Center for Education 
Statistics (NCES) publicly available database. Figure 2 shows that schools in the i-Ready 
Instruction and comparison groups share a relatively similar urbanicity distribution; though the /- 
Ready Instruction group includes a higher percentage of students from suburban schools than 
the comparison, and the comparison group includes a higher percentage of students from cities. 


Students’ School Urbanicity 
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Figure 2. Students’ school urbanicity for final matched i-Ready Instruction and 
comparison samples 


Baseline Equivalence 


Once our analytic samples were identified, we used our baseline difference model to estimate 
the adjusted mean differences between our i-Ready Instruction and comparison groups of 
students at each grade level. We converted the estimated baseline difference between students 
in the two groups to an effect size to evaluate baseline equivalence for each of the six analytic 
samples. For all six samples, Hedges’ g was much smaller than the WWC required threshold of 
0.25 (see Table 1), so we determined the groups were baseline equivalent (WWC, 2017b). 
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Table 1. Mathematics Baseline Equivalence Statistics for i-Ready Instruction (Treatment) 


and Comparison Groups by Grade 
| agp Mean "Effect 


‘i-Ready Instruction _ 4,306 350.37, 2174, 3.10, 
Comparison 4,178 347.27 23.21 
. ‘i-Ready Instruction | 12,534 379.29 24.03 0.34 0.01 
Comparison | 10,974 | 378.95 © 24.65 > 
i-Ready Instruction 15,547 406.36 24.44 0.25 0.01 
. Comparison 13,747 406.11 24.25 
“i-Ready Instruction «16,077, Ss «427.91, 25.59 «0.08 «0.00 
: Comparison 14,165 427.94 25.54 
i-Ready Instruction 17,337 450.77 28.13 -0.62 -0.02 
Comparison 15,536 451.39 28.13 
i-Ready Instruction «18,418 +=» 466.23 29.46] -2.27 -0.08 
"Comparison 16,462 = 468.50 29.31 | | 


Notes: SD = standard deviation of i-Ready scores, Adj hican Diff = adjusted mean difference 
between /-Ready Instruction and comparison groups, and Effect Size = Hedge’s g. 


Impact Analysis Results 


After confirming our matched samples were baseline equivalent at each grade, we estimated 
the impact of /-Ready Instruction on student achievement using the analytic model described 
above with spring 2019 i-Ready Diagnostic scores as the outcome. Analyses were conducted 
separately for each grade. This section describes the results of the analysis. Full information on 
the model results, including student- and school-level covariate parameters, are presented in 
Appendix B. 


In addition to estimating the impact of i-Ready Instruction, we also examined three model 
assumptions associated with two-level HLM—residual normality, independence, and 
homoscedasticity—using the MIXED_DX macro in SAS (Bell, Smiley, Ene, & Blue, 2014). No 
major violations were found. Additional details regarding the assumption checks are available in 
Appendix C. 


Table 2 contains the impact model results by grade for mathematics spring /-Ready Diagnostic 
scores. For all grade levels, the adjusted mean differences were positive, indicating the i-Ready 
Instruction group earned higher scores than the matched comparison group. All the mean 
differences were Statistically significant (a = .05) with Hedge’s g effect sizes ranging from 0.11 
to 0.31. These effect sizes are promising for an education intervention. Lipsey et al. (2012) 
suggested an effect size of 0.25 is large for an education intervention, and those of 0.15 or 
higher could be considered modest. Thus, the effect for Kindergarten (0.31) would be 
considered large by this standard, and the effect size for grade 1 (0.15) considered modest. 
Kraft (2019) notes traditional guidelines, including those reported by Lipsey, are often too rigid 
for the realities of education interventions. He specifies effect size ranges of 0.03—0.17 are 
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typical of education interventions and that these often represent a meaningful effect. All effect 
sizes in this study are towards the upper end of that range except for kindergarten, which 
exceeds it. He suggests effect sizes should be considered in conjunction with all aspects of an 
intervention, including the magnitude of the treatment contrast and costs. 


Table 2 also provides the intra-class correlations (ICCs) by grade. The ICCs measure the 
proportion of the variance between schools—that is, how much of the variance in mathematics /- 
Ready Diagnostic scores can be explained by school-level differences. The ICCs range from 
0.20 (grade 2) to 0.25 (Kindergarten). This suggests the majority of variance is due to factors 
other than school-level differences; however, we prefer ICCs to be below .20, and this was not 
the case for the six grades examined. The slightly elevated ICCs may be impacted by the 
variation in implementation methods and our decision to model implementation at the student- 
level. This finding will assist in future efforts for identifying the most appropriate unit of 
assignment to account for these variations. 
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Table 2. Impact Analysis Results for i-Ready Instruction (Treatment) and Comparison Groups for Mathematics Student 
Achievement by Grade 


Xo lm terete Effect 


Students 


Pim Size 


-K _ -Ready Instruction 0.25 4,306 387.91 23,48 759 <.0001 0.31 

Comparison 4,178 380.32 25.71 (0.87) 

1 i-Ready Instruction 0.21 12,534 414.24 25.58 3.77 <.0001 0.15 
Comparison 10,974 410.47 26.26 (0.49) 

2 i-Ready Instruction 0.20 15,547 437.54 25.78 3.54 <.0001 0.14 
Comparison 13,747 434.00 25.91 (0.41) 

2 i-Ready Instruction 0.21 16,077 460.24 28,13 3.47 <.0001 0.12 
Comparison 14,165 456.77 28.52 (0.41) 

4 i-Ready Instruction 0.23 17,337 479.90 30.40 3.72 <.0001 0.12 
Comparison 15,536 476.18 30.53 (0.41) 

5 i-Ready Instruction 0.22 18,418 490.72 31.24 3.39 <.0001 0.11 
Comparison 16,462 487.33 31.30 (0.38) 


Notes: ICC = intraclass correlation, SD = standard deviation of i-Ready scores, Adj Mean Diff = adjusted mean difference between 
i-Ready Instruction and comparison groups, SE = standard error of the adjusted mean difference, and Effect Size = Hedge’s g. 
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Summary and Discussion 


At all grades, impact analyses suggest that elementary school students who use /-Ready 
Instruction with fidelity have higher achievement in mathematics when compared to students 
who did not use /-Ready Instruction. At each grade, students in the /-Ready Instruction group 
had a statistically significantly higher mathematics i-Ready Diagnostic score than did students in 
a matched comparison group. 


The effect sizes provided additional evidence /-Ready Instruction is beneficial for improving 
student mathematics. Recent research (Kraft, 2019) suggests education interventions typically 
attain effects ranging from 0.03 to 0.17. Our effect sizes for grades 1—5 fell at the upper end of 
this range, and the effect size for Kindergarten exceeded it (0.31). Kraft (2019) notes one should 
consider various factors when interpreting effect sizes, including a program’s cost relative to its 
benefits and the size of the treatment contrast. For example, we note that /-Ready Instruction is 
a supplemental intervention that requires only 12 to 18 weeks of 30 minutes or more per week 
during a school year to be considered implemented with fidelity at the student-level. In addition, 
because /-Ready Instruction is not a full curriculum and there are likely many similarities 
between what else students are exposed to whether in the /-Ready Instruction group or 
comparison group, we believe the contrast between our treatment and comparison group is 
likely minimal. Similarly, it is possible some students in our comparison group were exposed to 
interventions like /-Ready Instruction. Thus, given the required effort for using i-Ready 
Instruction with fidelity is relatively low, and the contrast between the /-Ready Instruction and 
comparison group small compared to a more involved intervention or curricular program, we feel 
confident that our effect sizes are meaningful. 


Kraft (2019) also points out that the U.S. education system is decentralized, and implementation 
procedures are ultimately controlled by local schools and/or teachers. As a QED, this study did 
not attempt to control for curriculum, supplemental resources, or classroom structure. Students 
in both groups were not participants in a research study but rather they were actual customers 
and everyday users, and /-Ready Instruction was carried out in real-world conditions. We may 
have found even larger effect sizes had the study been conducted under more controlled 
circumstances. Impacts are typically greater for studies that aim for ideal or close to ideal 
implementation and less for studies that examine real-world implementation. Thus, the fact we 
were able to find significant findings for all grade levels despite the lack of controls is promising. 


We conducted this study differently from a past study using 2017-18 data by considering the 
unit of assignment to be the student instead of the school. Additionally, we used 2018-19 data 
to take advantage of the most recent available information. Despite these key differences, our 
results were highly consistent—both studies found positive, significant results in favor of i-Ready 
Instruction, and both studies had the largest effect sizes at the early grades. This replication 
provides confidence that students using /-Ready Instruction in conjunction with the i-Ready 
Diagnostic show greater mathematics achievement compared to a comparison group using /- 
Ready Diagnostic only. 


Our study was conducted as a rigorous QED to meet the current standards described by the 
WWC (WWC, 2017) to achieve a rating of Meets WWC Group Design Standards with 
Reservations. In addition, because we found statistically significant positive effects for all grades, 
this study meets the guidelines set forth by ESSA for a Level 2 (or Moderate) rating for evidence- 
based research (U.S. Department of Education, 2016). 
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Limitations and Implications for Future Studies 


This study provides strong evidence supporting mathematics /-Ready Instruction use for 
students. Through our long-standing relationship with Curriculum Associates and multiple 
impact evaluations, including the current study, we have developed recommendations for the 
foci of future studies that may provide additional evidence to support the impact of i-Ready 
Instruction. 


First, our |CCs were at or slightly above 0.20 for all grades, suggesting school differences may 
be important for matching and estimating treatment effects. However, the data also revealed 
large variations in how many students at a given school or grade within a school used /-Ready 
Instruction with fidelity. Future studies may look to explore the grade or classroom as the unit of 
assignment. We also recommend Curriculum Associates collect information directly from 
schools to understand their intended implementation so this information can be incorporated 
into sample selection and analytic models. 


Second, we note our study was a QED with the typical limitations, including a lack of information 
on implementation decisions made at each school and within each classroom. We recommend 
randomized control trials (RCTs) in the future even if only a small sample of schools and 
students is included. We also suggest including only one district to allow greater control on 
implementation. 


Finally, our treatment group was compared to a matched comparison group using the /-Ready 
Diagnostic. It is possible that use of i-Ready Diagnostic itself increases student achievement. 
However, the design of this study did not allow for an estimation of that impact. Further, use of 
the i-Ready Diagnostic only schools and students as a comparison group may have attenuated 
the effects of i-Ready Instruction use had this treatment group been compared to a “business- 
as-usual” comparison group. Future studies might examine the impact of /-Ready Instruction 
using a set of comparison schools and students not implementing any Curriculum Associates 
products. This would require an external achievement measure, potentially a state assessment, 
as the baseline and outcome measure. 


Quality Control Procedures 


We employed various quality control checks throughout the data cleaning, analysis, and 
reporting processes. HumRRO, Curriculum Associates, and Century Analytics worked together 
to identify a rigorous methodology based on implementation of i-Ready Instruction with fidelity, 
the WWC 4.0 standards, and ESSA Level 2 guidelines. 


Rules for identifying treatment and comparison groups were determined through collaboration 
between the three study partners. Curriculum Associates provided information on the various 
components of i-Ready Instruction and the frequency for which it should be used for 
implementation with fidelity. They also provided /-Ready Diagnostic and Instruction data to allow 
HumRRO and Century Analytics to empirically examine the extent to which these 
recommendations were followed by /-Ready Instruction schools. These discussions led to 
treatment and comparison group criteria in which all partners were confident. 


Data analysis work was completed collaboratively by HumRRO and Century Analytics. Century 
Analytics and HumRRO independently conducted matching and HLM analyses for each grade. 
The researchers reviewed results against each other and worked out any discrepancies. All 
results reported in this study were verified by researchers from both organizations. 
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Appendix A. /-Ready Instruction Theory of Action 
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. d , The i-Ready Diagnostic is an adaptive assessment that assesses students on relevant skills in a challenging and engaging way, capturing insight about 
Ready Diagnostic student learning down to the subskill level. Teachers are provided with precise, actionable data and instructional recommendations to more seamlessly 


differentiate dassroom instruction according to their students’ needs, saving teachers valuable time. This allows teachers to deliver more impactful 
instruction to increase student growth and proficiency. 


— seaoeaenomeancee 


The /-Ready Diagnostic is an adaptive assessment that assesses students 
three times 2 year on relevant skills in a challenging and engaging way, 

Capturing insight about individual student math and/or reading strengths 
and needs down to the sub-skill level. 


The following implementation program components help to maximally 
leverage the +Ready Diagnostic scores for differentiated instruction: 


OD Students access their customized /-Ready dashboard to view their data, 
performance, and progress. 


O Teachers attend Professional Development sessions to acquire } Ready 
skis and concepts. 


O Teachers ensure that students’ /Ready Diagnostic scores are valid and 
reliable by: 
© Adequately preparing students before taking the + Ready Diagnostic 
a le it tls 


© Ptanning to retest students with abnormal test results (ex: red rush 
flags) 

© Monitoring and observing students during the /-Ready Diagnostic 
admintstration 


O Teachers access the Ready dashboard and reports for: 

Precise and actionable performance and growth data 

© Gear tools such as student can dos and next steps for instruction, 
inctuding grade placement levets that highlight student needs down 
to the sub-skill level 

© Student groups based on similar instructional needs 

© Typkal and Stretch Growth values for each student 

Monitoring student growth over time 


O Teachers can display class goals, performance, and progress through data 
walls or other methods. 


O School and district leaders provide necessary system support: serving 2s 
instructional leaders, supporting teachers with implementation, ensuring 
the required technology is in place, setting appropriate schedules to 
administer the }Ready Dingnostic, clearly communicating those 
admanistration windows, and accessing reports to view student and class 
Gata to better understand resource needs. 


Appendix B. Impact HLM Coefficients 


Table B.1. HLM Results for Kindergarten Mathematics 


Student-Level Covariates 


Treatment Group Membership 7.59 0.87 8.72 | <0.001 5.88 9.30 
Fall 2018 Mathematics i-Ready Grand 
Mean Centered 0.63 0.01 69.34  <0.001 0.61 0.65 
Student-Level Stratum 
Female, ELL = 0, SpEd = 0, EcDis = 0 8.93 2.79 3.20 0.001 3.45 14.40 
Female, ELL = 1, SpEd = 0, EcDis = 0 451 2.96 1.52 0.128 -1.30 10.32 
Female, ELL = 0, SpEd = 1, EcDis = 0 5.18 3.10 1.67 0.095 -0.90 11.26 
Female, ELL = 0, SpEd = 0, EcDis = 1 5.89 2.81 2.10 0.036 0.38 11.39 
Female, ELL = 1, SpEd = 1, EcDis = 0 4.47 6.05 0.74 0.460 -7.39 16.32 
Female, ELL = 0, SpEd = 1, EcDis = 1 -0.26 | 3.24 -0.08 0.935 -6.61 6.08 
Female, ELL = 1, SpEd = 0, EcDis = 1 5.09 2.91 1.75 0.081 -0.62 10.80 
Female, ELL = 1, SpEd = 1, EcDis = 1 441 435 1.01 0.310 -4.11 12.93 
Male, ELL = 0, SpEd = 0, EcDis = 0 10.16 2.80 3.63 <0.001 4.68 15.64 
Male, ELL = 1, SpEd = 0, EcDis = 0 3.06 3.06 1.00 0.316 -2.93 9.06 
Male, ELL = 0, SpEd = 1, EcDis = 0 6.26 2.94 2.13 0.033 0.50 12.02 
Male, ELL = 0, SpEd = 0, EcDis = 1 7.39 2.82 2.62 0.009 1.86 12.91 
Male, ELL = 1, SpEd = 1, EcDis = 0 10.05 4.16 2.42 0.016 1.90 18.20 
Male, ELL = 0, SpEd = 1, EcDis = 1 163 3.00 0.54 0.586 -4.24 7.50 
Male, ELL = 1, SpEd = 0, EcDis = 1 5.36 2.93 1.83 0.067 -0.38 11.11 
School-Level Covariates 
Charter or Magnet Designation 6.81 1.44 4.72 | <0.001 3.98 9.64 
Traditional Elementary School 1.11 0.96 1.16 0.245 -0.77 2.99 
Percent non-white students -0.07 0.02 -2.92 0.003 -0.11 -0.02 
Percent FRL students -0.06 0.02 -2.47 0.014 -0.10 -0.01 
Locale — Suburban -4.03 1.68 -2.40 0.017 -7.33 -0.73 
Locale — Rural -3.04 1.03 -2.96 0.003 -5.05 -1.03 
Locale — City -6.09 | 2.30 -2.65 0.008 -10.60 -1.59 
Intercept 


Intercept 379.17 3.19 | 118.83 | <0.001 372.91 385.42 
Note. Stratum 16 and Locale — Town were used as reference groups in the model. 
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Table B.2. HLM Results for First Grade Mathematics 


Student-Level Covariates 


Treatment Group Membership 3.77 0.49 7.72 <0.001 2.82 4.73 
Fall 2018 Mathematics i-Ready Grand 
Mean Centered 0.76 0.00 | 155.22 <0.001 0.75 0.76 
Student-Level Stratum 
Female, ELL = 0, SpEd = 0, EcDis = 0 3.81 1.55 2.45 0.014 0.77 6.86 
Female, ELL = 1, SpEd = 0, EcDis = 0 2.19 1.69 1.29 0.195 -1.12 5.49 
Female, ELL = 0, SpEd = 1, EcDis = 0 -0.67 1.73 -0.39 0.698 -4.05 2.71 
Female, ELL = 0, SpEd = 0, EcDis = 1 1.18 1.55 0.76 0.448 -1.86 4.21 
Female, ELL = 1, SpEd = 1, EcDis = 0 -5.86 3.03 -1.93 0.053 -11.80 0.08 
Female, ELL = 0, SpEd = 1, EcDis = 1 -4,32 1.78 -2.43 0.015 -7.81 -0.83 
Female, ELL = 1, SpEd = 0, EcDis = 1 -0.74 1.61 -0.46 0.644 -3.90 2.41 
Female, ELL = 1, SpEd = 1, EcDis = 1 -4,31 2.64 -1.63 0.102 -9.49 0.86 
Male, ELL = 0, SpEd = 0, EcDis = 0 5.70 1.56 3.67 <0.001 2.65 8.75 
Male, ELL = 1, SpEd = 0, EcDis = 0 3.43 1.71 2.01 0.045 0.08 6.79 
Male, ELL = 0, SpEd = 1, EcDis = 0 2.69 1.62 1.65 0.098 -0.50 5.87 
Male, ELL = 0, SpEd = 0, EcDis = 1 2.56 1.55 1.65 0.099 -0.48 5.60 
Male, ELL = 1, SpEd = 1, EcDis = 0 1.46 2.38 0.61 0.539 -3.20 6.11 
Male, ELL = 0, SpEd = 1, EcDis = 1 -0.38 1.68 -0.23 0.821 -3.67 2.91 
Male, ELL = 1, SpEd = 0, EcDis = 1 2.23 1.62 1.38 0.169 -0.95 5,42 
School-Level Covariates 
Charter or Magnet Designation 0.72 0.78 0.92 0.359 -0.81 2.25 
Traditional Elementary School 1.48 0.49 3.03 0.002 0.52 2.44 
Percent non-white students -0.01 0.01 -0.78 0.433 -0.03 0.01 
Percent FRL students -0.08 0.01 -7.21 <0.001 -0.10 -0.06 
Locale — Suburban 0.32 0.83 0.39 0.700 -1.31 1.95 
Locale — Rural 0.20 0.52 0.39 0.698 -0.82 1.23 
Locale — City 0.92 1.04 0.88 0.377 -1.12 2.95 
Intercept 


Intercept | 410.99 1.73 | 237.98 <0.001 407.60 414.37 


Note. Stratum 16 and Locale — Town were used as reference groups in the model. 
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Table B.3. HLM Results for Second Grade Mathematics 


Student-Level Covariates 


Treatment Group Membership 3.54 0.41 8.56 <0.001 2.73 4.35 
Fall 2018 Mathematics i-Ready Grand 
Mean Centered 0.79 0.00 192.54 <0.001 0.78 0.80 
Student-Level Stratum 
Female, ELL = 0, SpEd = 0, EcDis = 0 6.75 1.18 5.71 <0.001 4.43 9.06 
Female, ELL = 1, SpEd = 0, EcDis = 0 5,21 1.31 3.98 <0.001 2.65 7.78 
Female, ELL = 0, SpEd = 1, EcDis = 0 1.81 1.32 1.37 0.171 -0.78 4.39 
Female, ELL = 0, SpEd = 0, EcDis = 1 4.07 1.18 3.46 0.001 1.77 6.38 
Female, ELL = 1, SpEd = 1, EcDis = 0 -1.30 2.82 -0.46 0.645 -6.82 4.22 
Female, ELL = 0, SpEd = 1, EcDis = 1 -1.24 1.38 -0.90 0.368 -3.95 1.46 
Female, ELL = 1, SpEd = 0, EcDis = 1 4.37 1.25 3.50 <0.001 1.92 6.81 
Female, ELL = 1, SpEd = 1, EcDis = 1 -5.78 1.95 -2.97 0.003 -9.60 -1.96 
Male, ELL = 0, SpEd = 0, EcDis = 0 6.93 1.18 5.85 <0.001 4.61 9.25 
Male, ELL = 1, SpEd = 0, EcDis = 0 6.60 1.31 5.03 <0.001 4.03 9.16 
Male, ELL = 0, SpEd = 1, EcDis = 0 3.71 1.25 2.98 0.003 1.27 6.15 
Male, ELL = 0, SpEd = 0, EcDis = 1 5,11 1.18 4.33 <0.001 2.80 7.43 
Male, ELL = 1, SpEd = 1, EcDis = 0 1.98 1.89 1.05 0.295 -1.72 5.69 
Male, ELL = 0, SpEd = 1, EcDis = 1 0.43 1.27 0.34 0.737 -2.07 2.93 
Male, ELL = 1, SpEd = 0, EcDis = 1 4.45 1.26 3.53 <0.001 1.98 6.92 
School-Level Covariates 
Charter or Magnet Designation 0.17 0.67 0.26 0.793 -1.13 1.48 
Traditional Elementary School 1.34 0.40 3.31 0.001 0.54 2.13 
Percent non-white students 0.00 0.01 -0.31 0.760 -0.02 0.02 
Percent FRL students -0.04 0.01 -4.38 <0.001 -0.06 -0.02 
Locale — Suburban 1.32 0.68 1.94 0.053 -0.02 2.66 
Locale — Rural 1.40 0.44 3.21 0.001 0.54 2.25 
Locale — City 1.55 0.83 1.88 0.061 -0.07 3.17 
Intercept 


Intercept | 429.20 134 319.91 <0.001 | 426.57 431.83 


Note. Stratum 16 and Locale — Town were used as reference groups in the model. 
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Table B.4. HLM Results for Third Grade Mathematics 


Student-Level Covariates 


Treatment Group Membership 3.47 0.41 8.43 <0.001 2.66 4.27 
Fall 2018 Mathematics i-Ready Grand 
Mean Centered 0.85 0.00 | 211.27 <0.001 0.84 0.86 


Student-Level Stratum 
Female, ELL = 0, SpEd = 0, EcDis = 0 9.72 1.20 8.11 <0.001 7.37 12.07 
Female, ELL = 1, SpEd = 0, EcDis = 0 8.94 1.36 6.58 <0.001 6.28 11.60 


Female, ELL = 0, SpEd = 1, EcDis = 0 5.52 1.31 4.23 <0.001 2.96 8.09 
Female, ELL = 0, SpEd = 0, EcDis = 1 6.66 1.19 5.59 <0.001 4.32 8.99 
Female, ELL = 1, SpEd = 1, EcDis = 0 3.48 2.39 1.46 0.146 -1.21 8.18 
Female, ELL = 0, SpEd = 1, EcDis = 1 -0.11 1.36 -0.08 0.935 -2.77 2.55 
Female, ELL = 1, SpEd = 0, EcDis = 1 6.79 1.25 5.43 <0.001 4.34 9.24 
Female, ELL = 1, SpEd = 1, EcDis = 1 0.05 1.77 0.03 0.977 -3.42 3.52 


Male, ELL = 0, SpEd = 0, EcDis = 0 11.03 1.20 9.19 <0.001 8.68 13.39 
Male, ELL = 1, SpEd = 0, EcDis = 0 11.43 1.34 8.56 <0.001 8.82 14.05 


Male, ELL = 0, SpEd = 1, EcDis = 0 6.52 1.25 5.22 <0.001 4.08 8.97 
Male, ELL = 0, SpEd = 0, EcDis = 1 8.18 1.20 6.84 <0.001 5.84 10.53 
Male, ELL = 1, SpEd = 1, EcDis = 0 5.79 1.99 2.91 0.004 1.88 9.69 
Male, ELL = 0, SpEd = 1, EcDis = 1 2.92 1.28 2.29 0.022 0.42 5.43 


Male, ELL = 1, SpEd = 0, EcDis = 1 8.59 1.26 6.83 <0.001 6.13 11.05 
School-Level Covariates 


Charter or Magnet Designation 0.12 0.66 0.18 0.857 -1.18 1.42 
Traditional Elementary School 0.66 0.41 1.63 0.102 -0.13 1.46 
Percent non-white students 0.02 0.01 1.87 0.062 0.00 0.03 
Percent FRL students -0.06 0.01 -7.11 <0.001 -0.08 -0.05 
Locale — Suburban 1.20 0.69 1.74 0.082 -0.15 2.55 
Locale — Rural 0.50 0.43 1.16 0.247 -0.35 1.34 
Locale — City -0.86 0.80 -1.09 0.278 -2.43 0.70 
Intercept 


Intercept | 450.30 1.35 | 334.34 <0.001 | 447.66 452.94 


Note. Stratum 16 and Locale — Town were used as reference groups in the model. 
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Table B.5. HLM Results for Fourth Grade Mathematics 


Student-Level Covariates 


Treatment Group Membership 3.72 0.41 9.13 <0.001 2.92 4.52 
Fall 2018 Mathematics i-Ready Grand 

Mean Centered 0.84 0.00 234.86 <0.001 0.84 0.85 

Student-Level Stratum 
Female, ELL = 0, SpEd = 0, EcDis = 0 8.34 1.10 7.56 <0.001 6.18 10.51 
Female, ELL = 1, SpEd = 0, EcDis = 0 4.75 1.30 3.65 <0.001 2.20 7.29 
Female, ELL = 0, SpEd = 1, EcDis = 0 1.14 1.22 0.94 0.349 -1.25 3.54 
Female, ELL = 0, SpEd = 0, EcDis = 1 6.35 1.10 5.79 <0.001 4.20 8.50 
Female, ELL = 1, SpEd = 1, EcDis = 0 -1.62 2.64 -0.61 0.540 -6.80 3.56 
Female, ELL = 0, SpEd = 1, EcDis = 1 -2.19 1.22 -1.79 0.073 -4.58 0.21 
Female, ELL = 1, SpEd = 0, EcDis = 1 4.66 1.17 3.98 <0.001 2.36 6.95 
Female, ELL = 1, SpEd = 1, EcDis = 1 -5.16 1.78 -2.89 0.004 -8.66 -1.66 


Male, ELL = 0, SpEd = 0, EcDis = 0 9.41 1.11 8.51 <0.001 7.24 11.57 
Male, ELL = 1, SpEd = 0, EcDis = 0 9.44 1.28 7.36 <0.001 6.92 11.96 


Male, ELL = 0, SpEd = 1, EcDis = 0 4.30 1.17 3.69 <0.001 2.01 6.59 
Male, ELL = 0, SpEd = 0, EcDis = 1 6.90 1.10 6.27 <0.001 4.74 9.05 
Male, ELL = 1, SpEd = 1, EcDis = 0 2.01 2.03 0.99 0.321 -1.96 5,99 
Male, ELL = 0, SpEd = 1, EcDis = 1 1.09 1.18 0.92 0.356 -1.22 3.40 
Male, ELL = 1, SpEd = 0, EcDis = 1 6.05 1.17 5.18 <0.001 3.76 8.33 

School-Level Covariates 
Charter or Magnet Designation -0.41 0.70 -0.58 0.560 -1.78 0.96 
Traditional Elementary School -0.70 0.42 -1.67 0.094 -1.53 0.12 
Percent non-white students 0.02 0.01 2.65 0.008 0.01 0.04 
Percent FRL students -0.06 0.01 -6.95 <0.001 -0.08 -0.05 
Locale — Suburban 2.25 0.71 3.15 0.002 0.85 3.65 
Locale — Rural 1.62 0.45 3.61 <0.001 0.74 2.50 
Locale — City 0.98 0.81 1.21 0.227 -0.61 2.58 

Intercept 


Intercept | 470.79 1.28 368.15 <0.001 | 468.28 473.29 


Note. Stratum 16 and Locale — Town were used as reference groups in the model. 


Impact Evaluation of Mathematics i-Ready Instruction for Elementary Grades using 2018-19 Data B-5 


Table B.6. HLM Results for Fifth Grade Mathematics 


Student-Level Covariates 


Treatment Group Membership 3.39 0.38 9.03 <0.001 2.65 4.13 
Fall 2018 Mathematics i-Ready Grand 
Mean Centered 0.87 0.00 259.14 <0.001 0.86 0.87 
Student-Level Stratum 
Female, ELL = 0, SpEd = 0, EcDis = 0 6.18 0.92 6.75 <0.001 4.38 7.97 
Female, ELL = 1, SpEd = 0, EcDis = 0 5,52 1.19 4.63 <0.001 3.19 7.86 
Female, ELL = 0, SpEd = 1, EcDis = 0 0.72 1.04 0.69 0.487 -1.32 2.77 
Female, ELL = 0, SpEd = 0, EcDis = 1 4.33 0.91 4.79 <0.001 2.56 6.11 
Female, ELL = 1, SpEd = 1, EcDis = 0 1.30 2.42 0.54 0.591 -3.44 6.03 
Female, ELL = 0, SpEd = 1, EcDis = 1 0.70 1.06 0.66 0.512 -1.39 2.78 
Female, ELL = 1, SpEd = 0, EcDis = 1 2.54 1.00 2.55 0.011 0.59 4.50 
Female, ELL = 1, SpEd = 1, EcDis = 1 0.24 1.54 0.16 0.874 -2.78 3.26 
Male, ELL = 0, SpEd = 0, EcDis = 0 7.07 0.92 7.70 <0.001 5,27 8.87 
Male, ELL = 1, SpEd = 0, EcDis = 0 5,55 1.16 4.80 <0.001 3.28 7.82 
Male, ELL = 0, SpEd = 1, EcDis = 0 1.70 0.98 1.72 0.085 -0.23 3.63 
Male, ELL = 0, SpEd = 0, EcDis = 1 4.53 0.91 4.98 <0.001 2.74 6.31 
Male, ELL = 1, SpEd = 1, EcDis = 0 -0.42 1.85 -0.23 0.820 -4.04 3.20 
Male, ELL = 0, SpEd = 1, EcDis = 1 0.54 0.99 0.55 0.582 -1.39 2.48 
Male, ELL = 1, SpEd = 0, EcDis = 1 3.31 1.00 3.33 0.001 1.36 5,27 
School-Level Covariates 
Charter or Magnet Designation -0.18 0.65 -0.28 0.776 -1.45 1.08 
Traditional Elementary School 0.14 0.40 0.35 0.728 -0.64 0.92 
Percent non-white students 0.02 0.01 2.79 0.005 0.01 0.04 
Percent FRL students -0.06 0.01 -6.44 <0.001 -0.07 -0.04 
Locale — Suburban 1.83 0.67 2.72 0.007 0.51 3.15 
Locale — Rural 0.59 0.42 1.40 0.162 -0.24 1.41 
Locale — City 1.15 0.78 1.47 0.141 -0.38 2.68 
Intercept 


Intercept | 483.18 1.09 442.74 <0.001 | 481.04 485.32 


Note. Stratum 16 and Locale — Town were used as reference groups in the model. 
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Appendix C. Model Assumption Checks 


We examined three model assumptions associated with two-level HLM—residual normality, 
independence, and homoscedasticity—using the MIXED_DX macro in SAS (Bell, Smiley, Ene, 
& Blue, 2014) based on the analytic model for all six grade levels of this study. The MIXED_DX 
macro provides visual output including box-and-whisker plots, histograms, scatter plots, and 
summary tables to examine residual normality, linearity, homoscedasticity, and influential 
outliers. The macro provides this information for level 1 and level 2 residuals. 


We reviewed plots and summary tables at level 1 and level 2 for each grade level. These 
checks provided assurance that our analytic model was appropriate for our data. We examined 
histograms, box and whisker plots, and scatter plots to check residual normality. These plots 
supported that our residuals were generally normally distributed, particularly, the histograms of 
level 2 residuals produced highly symmetrical bell shape with little skewness or kurtosis. The 
level 1 residuals had some skewness but were close enough to normal to allow confidence. 
There was no evidence when examining level 1 residuals of clearly non-normal distributions 
such as a bi-modal distribution. Violation of assumptions of normality of level 1 residuals can 
adversely affect estimation of random effect coefficients and variance-covariance components, 
but typically will not adversely affect estimation of standard errors and, therefore, inferences 
regarding statistical significance. Given the primary purpose of the models was estimating 
treatment effects, the slight lack of normality of the level 1 residuals likely did not have 
implications for the findings presented in this report. 


Scatter plots of predicted values against residuals at level 1 and level 2 clearly illustrated 
random distributions and provided support for that assumptions regarding independence and 
homoscedasticity were not violated. 
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